Terminal apparatus for performing communication between remote locations

ABSTRACT

A system is provided that supports a remote proxy service in which, on behalf of a request source user, a request destination user goes to a location (local site) where the request source user cannot go, and performs what is instructed by the request source user. The system includes a first terminal apparatus used by the request source user and a second terminal apparatus used by the request destination user. The first terminal apparatus displays a moving image captured at a local site by the second terminal apparatus in real time. When the request source user selects an image included in the moving image displayed by the first terminal apparatus, the selected image is displayed on the second terminal apparatus. The request destination user can receive an instruction regarding the object appearing in the image while viewing the image selected by the request source user.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Phase Application under 35 U.S.C.371 of International Application No. PCT/JP2021/046184, filed on Dec.15, 2021, which claims priority to Japanese Patent Application No.2021-013970, filed on Jan. 29, 2021. The entire disclosures of the aboveapplications are expressly incorporated by reference herein.

BACKGROUND Technical Field

The present invention relates to a technique for supporting smoothcommunication between users at locations separated from each other.

Related Art

Accompanying the proliferation of communication networks such as theInternet and mobile communication networks and the improvement ofcommunication speeds in those communication networks, moving image datarepresenting a moving image (accompanying sound picked up by amicrophone) captured by a camera is transmitted and received insubstantially real time between terminal apparatuses used by users (inthe present application, the term “user” is not limited to humans, andincludes apparatuses capable of intellectual activity similar to that ofa human being, such as an apparatus having artificial intelligence) atlocations separated from each other, and by playing back those movingimages on the terminal apparatuses, it is possible to perform a videocall.

A service (hereinafter referred to as a “remote proxy service”) has beenproposed in which, using the above-described video call mechanism, auser who is a request source (hereinafter referred to as a “requestsource user”) finds a person (hereinafter referred to as a “requestdestination user”) to go on his or her behalf to a location (hereinafterreferred to as a “local site”) to which he or she cannot go himself orherself, and due to the request destination user who is at the localsite and the request source user who is not at the local site achievingcommunication with each other, the request source user has a simulatedexperience of going to the local site and has the request destinationuser do what the request source user wants to do at the local site(e.g., purchase of goods, business negotiation, etc.) on the requestsource user's behalf.

JP 2020-198474A is an example of a patent document that proposes amechanism for supporting the remote proxy service as described above.

In the remote proxy service, smooth communication may be difficultbetween the request source user and the request destination user. Forexample, the request source user may discover an object of interest inthe moving image of the local site being captured, and the requestsource user may want the request destination user to check what theobject is. In such a case, since the request source user does not knowwhat the object of interest is, it is not easy to convey the object tothe request destination user.

In view of the above-described circumstance, the present inventionprovides means for supporting smooth communication between users who areat locations separated from each other.

SUMMARY

The present invention provides, as a first aspect, a terminal apparatusincluding: a reception means for receiving moving image data transmittedin real time from a communication partner apparatus; a selected imageacquisition means for acquiring image identification data foridentifying a selected image among a plurality of images constituting amoving image represented by the moving image data received by thereception means; and a transmission means for transmitting the imageidentification data acquired by the selected image acquisition means, orimage data representing the image identified by the image identificationdata to the communication partner apparatus.

With the terminal apparatus according to the first aspect, it ispossible to notify the communication partner apparatus of the imageselected by the user among the images constituting the moving imagetransmitted from the communication partner apparatus. As a result, theuser can convey the object that he or she is interested in to the userwho is the communication partner by an image.

In the terminal apparatus according to the first aspect, it is alsopossible to adopt, as a second aspect, a configuration furtherincluding: a display instruction means for displaying the moving imagerepresented by the moving image data received by the reception means ona display apparatus; and an operation reception means for receiving auser operation on an image being displayed on the display apparatus, inwhich the selected image acquisition means acquires the imageidentification data for identifying the image that was being displayedby the display apparatus when the operation reception means received theuser operation on the moving image displayed by the display apparatus asimage identification data of a selected image or image identificationdata of a selected candidate image.

With the terminal apparatus according to the second aspect, if an imageof interest is displayed while playing back a moving image transmittedfrom the communication partner apparatus in substantially real time, theuser can select the image as an image to send to the communicationpartner apparatus or a candidate image to send to the communicationpartner apparatus by performing a predetermined operation.

In the terminal apparatus according to the first aspect, it is alsopossible to adopt, as a third aspect, a configuration further including:a storage means for storing at least a portion of the moving image datareceived by the reception means, the portion corresponding to a mostrecent past predetermined amount; an operation reception means forreceiving a user operation on an image being displayed on the displayapparatus; and a display instruction means for displaying the movingimage represented by the moving image data received by the receptionmeans on the display apparatus in real time, and displaying, on thedisplay apparatus, an image designated by an operation performed usingthe operation reception means among images constituting a past movingimage represented by the moving image data stored in the storage means,in which, if the operation reception means has received the useroperation for selecting the image designated by an operation performedusing the operation reception means, the image being displayed by thedisplay apparatus, the selected image acquisition means acquires theimage identification data of the selected image as image identificationdata of a selected image or image identification data of a selectedcandidate image.

With the terminal apparatus according to the third aspect, even if therewas an image of interest in a moving image that was transmitted from thecommunication partner apparatus and was played back in approximatelyreal time, but the operation for selecting the image was not performedat the timing at which the image was displayed, the user can select theimage later as an image to send to the communication partner apparatusor as a candidate for an image to send to the communication partnerapparatus.

In the terminal apparatus according to the second or third aspect, it isalso possible to adopt, as a fourth aspect, a configuration in which, ifthe image identification data of a selected candidate image has beenacquired by the selected image acquisition means, the displayinstruction means displays the image identified by the imageidentification data on the display apparatus as a selected candidateimage, and if the operation reception means has received the useroperation for selecting any one of one or more selected candidate imagesbeing displayed by the display apparatus, the selected image acquisitionmeans acquires the image identification data of the selected image asselected image identification data.

With the terminal apparatus according to the fourth aspect, the user canselect a candidate for an image to send to the communication partnerapparatus, and thereafter select an image to actually send to thecommunication partner apparatus from among one or more selectedcandidates.

In the terminal apparatus according to any one of the second to fourthaspects, it is also possible to adopt, as a fifth aspect, aconfiguration in which the operation reception means receives the useroperation for designating any region included in the selected imageidentified by the image identification data acquired by the selectedimage acquisition means, and the transmission means transmits regiondata indicating the region designated by the user to the communicationpartner apparatus.

With the terminal apparatus according to the fifth aspect, the user cannotify the user of the communication partner apparatus of the region ofinterest in the image transmitted to the communication partnerapparatus.

In any one of the second to fifth terminal apparatuses, it is alsopossible to adopt, as a sixth aspect, a configuration further includingan image recognition means for recognizing an object appearing in theselected image identified by the image identification data acquired bythe selected image acquisition means, in which the display instructionmeans displays, on the display apparatus, an image obtained by adding adisplay object indicating a result of recognition performed by the imagerecognition means to the selected image identified by the imageidentification data acquired by the selected image acquisition means.

With the terminal apparatus according to the sixth aspect, the user caneasily find out what appears in the image to send to the communicationpartner apparatus or the candidate image to send to the communicationpartner apparatus.

In any one of the second to sixth terminal apparatuses, it is alsopossible to adopt, as a seventh aspect, a configuration in which thedisplay instruction means displays a virtual operation button on thedisplay apparatus, and if the operation reception means has received theuser operation on the operation button, the transmission means transmitsmessage identification data for identifying a message corresponding tothe operation button or message data indicating the message, to thecommunication partner apparatus.

With the terminal apparatus according to the seventh aspect, the usercan accurately convey the message that he or she wants to convey to theuser of the communication partner apparatus by performing an operationon the operation button corresponding to the message.

In the terminal apparatus according to the seventh aspect, it is alsopossible to adopt, as an eighth aspect, a configuration in which, if theoperation reception means has received an operation of associating theimage selected by the user with the operation button, the transmissionmeans transmits the message identification data for identifying themessage corresponding to the operation button or the message dataindicating the message, in association with the image identificationdata acquired by the selected image acquisition means or the image datarepresenting the image identified by the image identification data, tothe communication partner apparatus.

With the terminal apparatus according to the eighth aspect, it ispossible to convey a message related to the image together with theimage to the user of the communication partner apparatus.

In the terminal apparatus according to the seventh or eighth aspect, itis also possible to adopt, as a ninth aspect, a configuration in whichthe message data is text data indicating a sentence written in alanguage used by a user of the communication partner apparatus or voicedata representing a sound of uttering the sentence.

With the terminal apparatus according to the ninth aspect, even if thelanguage used by the user is different from the language used by theuser of the communication partner apparatus, it is possible to convey amessage that the partner user can understand.

In the terminal apparatus according to any one of the seventh to ninthaspects, it is also possible to adopt, as a tenth aspect, aconfiguration in which the message data is data indicating a state ofthe user.

With the terminal apparatus according to the tenth aspect, the user canconvey his or her state to the user of the communication partnerapparatus.

In the terminal apparatus according to the seventh or eighth aspect, itis also possible to adopt, as an eleventh aspect, a configuration inwhich the message data is operation instruction data for instructing aphysical operation to the communication partner apparatus.

With the terminal apparatus according to the eleventh aspect, the usercan remotely control the physical operation of the communication partnerapparatus.

The present invention also provides, as a twelfth aspect, a terminalapparatus including: a moving image acquisition means for acquiringmoving image data representing a moving image captured by an imagecapture apparatus; a transmission means for transmitting the movingimage data acquired by the moving image acquisition means to acommunication partner apparatus in real time; a storage means forstoring at least a portion of the moving image data acquired by themoving image acquisition means, the portion corresponding to a mostrecent past predetermined amount; a reception means for receiving imageidentification data for identifying an image transmitted from thecommunication partner apparatus; and a display instruction means fordisplaying, on a display apparatus, the image identified by the imageidentification data received by the reception means among a plurality ofimages constituting the moving image represented by the moving imagedata stored in the storage means.

With the terminal apparatus according to the twelfth aspect, the usercan see and check the image selected by the user of the communicationpartner apparatus among the plurality of images constituting the movingimage transmitted to the communication partner apparatus.

In the terminal apparatus according to the twelfth aspect, it is alsopossible to adopt, as a thirteenth aspect, a configuration in which, ifthe reception means has received message identification data foridentifying a message transmitted in association with the imageidentification data or message data indicating the message from thecommunication partner apparatus, the display instruction means displays,on the display apparatus, the message identified by the messageidentification data or the message indicated by the message data,together with the image identified by the image identification data.

With the terminal apparatus according to the thirteenth aspect, the usercan find out the message of the user of the communication partnerapparatus regarding the image selected by the user of the communicationpartner apparatus.

In the terminal apparatus according to the thirteenth aspect, it is alsopossible to adopt, as a fourteenth aspect, a configuration furtherincluding a translation means for, if text data indicating a sentence ina language that is not a language used by a user of the terminalapparatus, acquiring text data indicating a sentence obtained bytranslating the sentence indicated by the text data into the languageused by the user, in which, if the message indicated by the message dataindicates a sentence in a language that is not the language used by theuser of the terminal apparatus, the display instruction means displays,on the display apparatus, a sentence obtained by translating thesentence indicated by the text data acquired by the translation means.

With the terminal apparatus according to the fourteenth aspect, the usercan read and understand a message from the user of the communicationpartner apparatus even if the user uses a language different from thatof the user of the communication partner apparatus.

In the terminal apparatus according to the thirteenth or fourteenthaspect, it is also possible to adopt, as a fifteenth aspect, aconfiguration further including: an interpretation means for, if textdata indicating a sentence in a language that is not a language used bya user of the terminal apparatus has been acquired, acquiring voice datarepresenting a sound of uttering a sentence obtained by translating thesentence indicated by the text data into the language used by the user;and a sound generation instruction means for, if the message indicatedby the message data indicates a sentence in a language that is not thelanguage used by the user of the terminal apparatus, causing a soundgeneration apparatus to generate the sound of uttering the sentenceobtained by translating the sentence represented by the voice dataacquired by the interpretation means.

With the terminal apparatus according to the fifteenth aspect, the usercan hear and understand the message from the user of the communicationpartner apparatus even if the user uses a language different from thatof the user of the communication partner apparatus.

In the terminal apparatus according to any one of the twelfth orfifteenth aspects, it is also possible to adopt, as a sixteenth aspect,a configuration further comprising: a position acquisition means foracquiring position data indicating a position of the terminal apparatus;and a map acquisition means for acquiring map data representing a map ofan area including the position of the terminal apparatus, in which thedisplay instruction means displays, on the display apparatus, an imageobtained by adding, to a map represented by the map data, a displayobject indicating the position indicated by the position datacorresponding to an image capture timing of the image identified by theimage identification data received by the reception means.

With the terminal apparatus according to the sixteenth aspect, the usercan know the location where the image selected by the user of thecommunication partner apparatus was captured.

In the terminal apparatus according to the sixteenth aspect, it is alsopossible to adopt, as a seventeenth aspect, a configuration in which themoving image acquisition means acquires, together with moving imagedata, image capture direction data indicating respective image capturedirections of a plurality of images constituting the moving imagerepresented by the moving image data, and the display instruction meansdisplays, on the display apparatus, an image obtained by adding, to themap represented by the map data, a display object indicating the imagecapture direction indicated by the image capture direction datacorresponding to the image capture timing of the image identified by theimage identification data received by the reception means.

With the terminal apparatus according to the seventeenth aspect, theuser can know the image capture direction of the image selected by theuser of the communication partner apparatus.

In the terminal apparatus according to any one of the twelfth toseventeenth aspects, it is also possible to adopt, as an eighteenthaspect, a configuration further including: a position acquisition meansfor acquiring position data indicating a position of the terminalapparatus; and a movement route acquisition means for acquiring movementroute data indicating a movement route from the position of the terminalapparatus indicated by the position data acquired most recently by theposition acquisition means, to a position indicated by the position datacorresponding to an image capture timing of the image identified by theimage identification data received by the reception means, in which thedisplay instruction means displays, on the display apparatus, a displayobject indicating the movement route indicated by the movement routedata.

With the terminal apparatus according to the eighteenth aspect, the usercan know the movement route for moving from the current position to thelocation where the image selected by the user of the communicationpartner apparatus was captured.

In the terminal apparatus according to the twelfth aspect, it is alsopossible to adopt, as a nineteenth aspect, a configuration furtherincluding an operation means for, if the reception means has receivedmessage identification data for identifying an operation instructioninstructing a physical operation, the operation instruction beingtransmitted in association with the image identification data, ormessage data indicating the operation instruction from the communicationpartner apparatus, performing a physical operation according to theoperation instruction identified by the message identification data orthe operation instruction indicated by the message data.

With the terminal apparatus according to the nineteenth aspect, the userof the communication partner apparatus can remotely control the terminalapparatus to cause the terminal apparatus to perform an operation thatthe user wants to perform at the local site.

In the terminal apparatus according to the twelfth aspect, it is alsopossible to adopt, as a twentieth aspect, a configuration furtherincluding: a position acquisition means for acquiring position dataindicating a position of the terminal apparatus; a movement routeacquisition means for acquiring movement route data indicating amovement route from the position of the terminal apparatus indicated bythe position data acquired most recently by the position acquisitionmeans, to a position indicated by the position data corresponding to animage capture timing of the image identified by the image identificationdata received by the reception means; and a movement means for movingaccording to the movement route indicated by the movement route data.

With the terminal apparatus according to the twentieth aspect, the userof the communication partner apparatus can move the terminal apparatusto the location where the image was captured by sending the imageselected by the user to the communication partner apparatus.

The present invention also provides, as a twenty-first aspect, a programfor causing a computer to execute: processing for receiving moving imagedata transmitted in real time from a communication partner apparatus;processing for acquiring image identification data for identifying aselected image among a plurality of images constituting the moving imagerepresented by the received moving image data; and processing fortransmitting the acquired image identification data or image datarepresenting the image identified by the image identification data tothe communication partner apparatus.

With the program according to the twenty-first aspect, the terminalapparatus according to the first aspect is realized by a computer.

The present invention also provides, as a twenty-second aspect, aprogram for causing a computer to execute: processing for acquiringmoving image data representing a moving image captured by an imagecapture apparatus; processing for transmitting the acquired moving imagedata to a communication partner apparatus in real time; processing forstoring at least a portion of the acquired moving image data, theportion corresponding to a most recent past predetermined amount;processing for receiving image identification data for identifying animage transmitted from the communication partner apparatus; andprocessing for displaying, on a display apparatus, the image identifiedby the received image identification data among a plurality of imagesconstituting the moving image represented by the stored moving imagedata.

With the program according to the twenty-second aspect, the terminalapparatus according to the twelfth aspect is realized by a computer.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing an overall configuration of a systemaccording to an embodiment.

FIG. 2 is a diagram showing a configuration of a computer for realizinga terminal apparatus according to an embodiment.

FIG. 3 is a diagram showing a functional configuration of a terminalapparatus according to an embodiment.

FIG. 4 is a diagram showing a screen displayed by a terminal apparatusused by a request source user according to an embodiment.

FIG. 5 is a diagram showing a screen displayed by a terminal apparatusused by a request destination user according to an embodiment.

DETAILED DESCRIPTION Embodiment

System 1 according to an embodiment of the present invention will bedescribed hereinafter. System 1 is a system that serves as a platformfor a remote proxy service. FIG. 1 is a diagram showing an overallconfiguration of system 1. System 1 includes terminal apparatus 11A,terminal apparatus 11B, management server apparatus 12, route searchserver apparatus 13, translation server apparatus 14, voice synthesisserver apparatus 15, and voice recognition server apparatus 16. Theseapparatuses constituting system 1 can communicate with each other vianetwork 9.

Terminal apparatus 11A is a terminal apparatus used by a request sourceuser of the remote proxy service. Terminal apparatus 11B is a terminalapparatus used by a request destination user of the remote proxyservice. In FIG. 1 , only one terminal apparatus 11A and one terminalapparatus 11B are shown, but in reality, the number of these terminalapparatuses varies depending on the number of users who use system 1.

Management server apparatus 12 is a server apparatus that manages theusers of system 1, supports matching between the request source user andthe request destination user, establishes a communication connectionbetween terminal apparatus 11A of the request source user and terminalapparatus 11B of the request destination user, and the like.

Route search server apparatus 13 is a server apparatus that, uponreceiving a request including position data indicating positions of adeparture location and a destination from any apparatus, specifies amovement route from the departure location to the destination inresponse to the request, and transmits movement route data indicatingthe specified movement route and map data representing a map of an areaincluding the specified movement route to the request source apparatus.

Translation server apparatus 14 is a server apparatus that, uponreceiving a request including text data indicating a sentence andlanguage designation data indicating a language into which the sentenceis to be translated from any apparatus, translates the text indicated bythe text data into the language indicated by the language designationdata in response to the request, and transmits text data indicating thetranslated sentence to the request source apparatus.

Voice synthesis server apparatus 15 is a server apparatus that, uponreceiving a request including text data indicating a sentence from anyapparatus, synthesizes a voice uttering the sentence indicated by thetext data in response to the request, and transmits voice datarepresenting the synthesized voice to the request source apparatus.

Voice recognition server apparatus 16 is a server apparatus that, uponreceiving a request including voice data representing a voice from anyapparatus, recognizes the voice represented by the voice data inresponse to the request, and transmits text data indicating therecognized sentence to the request source apparatus.

Each of management server apparatus 12, route search server apparatus13, translation server apparatus 14, voice synthesis server apparatus15, and voice recognition server apparatus 16 is realized by onecomputer or a plurality of computers operating in cooperation with eachother. The computer for realizing these server apparatuses is, forexample, a computer for a general server apparatus including aprocessor, a memory, and a communication interface.

Terminal apparatus 11A is realized due to a computer for a terminalapparatus such as a desktop PC (Personal Computer), a notebook PC, atablet PC, or a smart phone executing processing according to a programfor the terminal apparatus of the request source user according to thepresent embodiment. Since terminal apparatus 11A is used by the requestsource user, the computer for realizing terminal apparatus 11A does notnecessarily need to be portable. The computer for realizing terminalapparatus 11A includes or is connected to a microphone (an example of asound pickup apparatus) that picks up the voice of the request sourceuser, a keyboard (an example of an operation apparatus) that receives anoperation performed by the request source user, a display (an example ofa display apparatus) that displays various types of information to therequest source user, a speaker or the like (an example of a soundgeneration apparatus) that generates sound for the request source user,a camera (an example of an image capture apparatus) that captures theappearance and the like of the request source user, and a GNSS (GlobalNavigation Satellite System) unit (an example of a position measurementapparatus) that measures the position (e.g., latitude and longitude) ofthe computer on the Earth, and data can be transferred between thecomputer and these apparatuses.

Terminal apparatus 11B is realized by a computer for a terminalapparatus such as a tablet PC or a smartphone executing processingaccording to a program for the terminal apparatus of the requestdestination user according to the present embodiment. Since terminalapparatus 11B is used by a request destination user who needs to move ata local site, the computer for realizing terminal apparatus 11B needs tobe portable. The computer for realizing terminal apparatus 11B includesor is connected to a microphone (an example of a sound pickup apparatus)that picks up the voice of the request destination user, a touch panelor the like (an example of an operation apparatus) that receives theoperation of the request destination user, a display (an example of adisplay apparatus) that displays various types of information to therequest destination user, a speaker or the like (an example of a soundgeneration apparatus) that generates sound for the request destinationuser, a camera (an example of an image capture apparatus) that capturesan image of the surrounding area and the like of the request destinationuser, a GNSS (Global Navigation Satellite System) unit (an example of aposition measurement apparatus) that measures the position (e.g.,latitude and longitude) of the computer on the Earth, and an azimuthsensor (an example of an azimuth measurement apparatus) that measuresthe azimuth of the image capture direction of the camera of terminalapparatus 11B, and data can be transferred between the computer andthese apparatuses.

As described above, the computer for realizing terminal apparatus 11Aand the computer for realizing terminal apparatus 11B have differentrequired specifications, but in the following description, for the sakeof convenience, it is assumed that the computer for realizing terminalapparatus 11A and the computer for realizing terminal apparatus 11B havethe same specifications.

FIG. 2 is a diagram showing a configuration of computer 10 for realizingterminal apparatus 11A or terminal apparatus 11B. Computer 10 includesprocessor 101 that performs various types of data processing accordingto a program, memory 102 that stores various types of data including theprogram, communication interface 103 that performs data communicationwith another apparatus, touch screen 104 that serves as a userinterface, microphone 105 that picks up sound in the surrounding area,speaker 106 that generates sound, camera 107 that performs imagecapture, GNSS unit 108 that measures the position of computer 10, andazimuth sensor 109 that measures the azimuth of the image capturedirection of camera 107.

Note that touch screen 104 is, for example, a device in which a liquidcrystal display and a touch panel are laminated, and performs displaysof an image or the like to the user and accepts a touch operationperformed by the user on computer 10.

The functions of terminal apparatus 11A and the functions of terminalapparatus 11B are partially different, but many of these functions areheld in common. Accordingly, in the following description, forconvenience, the functional configuration included in terminal apparatus11A and the functional configuration included in terminal apparatus 11Bare assumed to be the same as each other. For example, terminalapparatus 11A need not include azimuth measurement apparatus 1105described below. Also, for example, reception means 111 included interminal apparatus 11A and reception means 111 included in terminalapparatus 11B receive different data. These differences will beclarified through description of the operation of terminal apparatus 11Aand terminal apparatus 11B, which will be described later with referenceto FIGS. 4 and 5 .

FIG. 3 is a diagram showing a functional configuration of terminalapparatus 11A or terminal apparatus 11B. Storage means 110 is realizedby memory 102 and stores various types of data. Reception means 111 isrealized by communication interface 103 and receives various types ofdata from other apparatuses.

Display apparatus 1101 is realized by touch screen 104 and displays animage. Display instruction means 112 is realized by processor 101 andcauses display apparatus 1101 to display an image. Sound generationapparatus 1102 is realized by speaker 106 and generates sound. Soundgeneration instruction means 113 is realized by processor 101 and causessound generation apparatus 1102 to generate sound.

Operation apparatus 1103 is realized by touch screen 104 and physicallyaccepts an operation performed by the user. Operation reception means114 is realized by processor 101 and receives the user's operationthrough software by specifying the content of the physical operationreceived by operation apparatus 1103. Selected image acquisition means115 is realized by processor 101 and acquires image identification datafor identifying an image to be selected from among a plurality of images(still images) constituting a moving image represented by moving imagedata received by reception means 111 from another apparatus.

Transmission means 116 is realized by communication interface 103, andtransmits various types of data to another apparatus.

Image capture apparatus 1104 is realized by camera 107, performs imagecapture, and generates moving image data representing a moving image.Moving image acquisition means 117 acquires the moving image datagenerated by image capture apparatus 1104. Azimuth measurement apparatus1105 is realized by azimuth sensor 109, measures the image capturedirection of image capture apparatus 1104, and generates image capturedirection data indicating the measurement result. Moving imageacquisition means 117 acquires the image capture direction datagenerated by azimuth measurement apparatus 1105 and associates it withthe moving image data acquired from image capture apparatus 1104.

Sound pickup apparatus 1106 is realized by microphone 105, picks upsound in the surrounding area, and mainly generates voice datarepresenting the user's voice. Voice acquisition means 118 acquires thevoice data generated by sound pickup apparatus 1106. Note that themoving image data acquired by moving image acquisition means 117 (movingimage data associated with the image capture direction data) and thevoice data acquired by voice acquisition means 118 are synthesized intosynchronized moving image data with audio (hereinafter referred tosimply as “moving image data”).

Position measurement apparatus 1107 is realized by GNSS unit 108,measures the position of the terminal apparatus, and generates positiondata indicating the measurement result. Position acquisition means 119is realized by processor 101 and acquires the position data generated byposition measurement apparatus 1107.

In the present embodiment, as an example, it is assumed that the requestsource user uses Japanese and the request destination user uses English.

In this case, transmission means 116 of terminal apparatus 11B transmitsa request including text data indicating a Japanese sentence receivedfrom terminal apparatus 11A to translation server apparatus 14, andreception means 111 of terminal apparatus 11B receives text dataindicating an English sentence transmitted from translation serverapparatus 14 as a response. That is, transmission means 116 andreception means 111 of terminal apparatus 11B serve as a translationmeans for, when text data indicating a sentence in a language that isnot the language used by the user of terminal apparatus 11B is acquired,acquiring text data indicating the sentence obtained by translating thesentence indicated by the text data into the language used by the user.

Also, transmission means 116 of terminal apparatus 11B transmits, tovoice synthesis server apparatus 15, a request including text dataindicating an English sentence received by reception means 111 ofterminal apparatus 11B from translation server apparatus 14, andreception means 111 of terminal apparatus 11B receives voice datarepresenting a voice uttering an English sentence transmitted from voicesynthesis server apparatus 15 as a response. That is, transmission means116 and reception means 111 of terminal apparatus 11B serve as aninterpretation means for, when text data indicating a sentence in alanguage that is not the language used by the user of terminal apparatus11B is acquired, acquiring voice data representing the sound of utteringthe sentence obtained by translating the sentence indicated by the textdata into the language used by the user.

Also, transmission means 116 of terminal apparatus 11B uses the mostrecent position data out of the position data acquired by positionacquisition means 119 as the position data of the departure location,and transmits a request including position data corresponding to theimage capture timing of the image identified by the image identificationdata received by reception means 111 of terminal apparatus 11B fromterminal apparatus 11A as position data of the destination to routesearch server apparatus 13, and reception means 111 of terminalapparatus 11B receives movement route data and map data transmitted fromroute search server apparatus 13 as a response.

That is, transmission means 116 and reception means 111 of terminalapparatus 11B serve as a map acquisition means for acquiring map datarepresenting a map of the area including the position of terminalapparatus 11B. Also, transmission means 116 and reception means 111 ofterminal apparatus 11B serve as a movement route acquisition means foracquiring movement route data indicating a movement route from theposition of terminal apparatus 11B indicated by the position data lastacquired by position acquisition means 119 of terminal apparatus 11B toa position indicated by the position data corresponding to the imagecapture timing of the image identified by the image identification datareceived by reception means 111 of terminal apparatus 11B.

Next, the operation of terminal apparatus 11A and terminal apparatus 11Bwill be described. In the following description, it is assumed thatterminal apparatus 11A and terminal apparatus 11B have established acommunication connection according to, for example, an instruction frommanagement server apparatus 12.

FIG. 4 is a diagram illustrating a screen (hereinafter referred to as“request source screen”) displayed by terminal apparatus 11A. In regionA01 of the request source screen, the moving image represented by themoving image data transmitted from terminal apparatus 11B to terminalapparatus 11A is displayed substantially in real time.

In the text box in region A02 of the request source screen, a Japanesesentence indicated by the text data received from translation serverapparatus 14 is displayed as a response to terminal apparatus 11Atransmitting, to translation server apparatus 14, a request includingthe message data received from terminal apparatus 11B.

When a sentence is displayed in the text box in region A02, a voiceuttering the sentence is generated from sound generation apparatus 1102of terminal apparatus 11A. The voice generated by sound generationapparatus 1102 is the voice represented by the voice data received fromvoice synthesis server apparatus 15 as a response obtained due toterminal apparatus 11A transmitting, to voice synthesis server apparatus15, a request including text data indicating the sentence displayed inthe text box of region A02. The request source user can make soundgeneration apparatus 1102 repeatedly generate sound by performing atouch operation on a “Playback” button in region A02.

In region A03 of the request source screen, an image selected by therequest source user from a plurality of images constituting the movingimage displayed in region A01 is displayed. By performing a touchoperation on region A01, the request source user can select an imagedisplayed in region A01 when the touch operation is performed. The imageselected in this manner is displayed in region A03.

Also, by dragging knob P02 on time bar P01 displayed on the moving imagein the left-right direction in region A01, the request source user candisplay, near knob P02, thumbnail P03 of an image of the timingcorresponding to the position in the left-right direction of knob P02among the images constituting the moving image displayed in region A01in the most recent past predetermined time. For this reason, storagemeans 110 of terminal apparatus 11A stores the portion of the movingimage data that corresponds to the most recent past predetermined time,the moving image data having been received from terminal apparatus 11Bby reception means 111. By performing a touch operation on thumbnailP03, the request source user can select the image corresponding tothumbnail P03 being displayed when the touch operation is performed. Theimage selected in this manner is displayed in region A03.

In region A04 of the request source screen, thumbnails of the imagesdragged and dropped from region A03 to region A04 by the request sourceuser among the images selected by the request source user and displayedin region A03 are displayed. The images for which thumbnails aredisplayed in region A04 are images selected as candidate images to beselected by the request source user. By dragging and dropping any one ormore thumbnails displayed in region A04 from region A04 to region A03,the request source user can display the images corresponding to thethumbnails in region A03.

A plurality of virtual operation buttons are displayed in region A05 ofthe request source screen. Of those operation buttons, the fouroperation buttons on the left side are operation buttons for the requestsource user to convey a fixed message to the request destination user.In order starting from the left, the four operation buttons shown inFIG. 4 are associated with the messages “What is that?”, “How much doesit cost?”, “Please show me it more clearly”, and “Please buy it”.

The request source user can register a plurality of sets of theseoperation buttons in terminal apparatus 11A in advance, and can switchthe set of these operation buttons by performing a touch operation onthe up-arrow button or the down arrow button displayed in region A05.The management server apparatus 12 stores various sets of operationbuttons, and the request source user can download a set includingoperation buttons that are used frequently in the remote proxy serviceto terminal apparatus 11A from among those sets. Also, the requestsource user can create a new set of operation buttons or change themessages associated with the operation buttons included in the setdownloaded from management server apparatus 12 to terminal apparatus11A.

When the request source user performs a touch operation on any of thefour operation buttons on the left side displayed in region A05,terminal apparatus 11A transmits message data indicating the messageassociated with the operation button on which the touch operation wasperformed to the terminal apparatus 11B.

When the request source user drags and drops the image displayed inregion A03 to any of the four operation buttons on the left sidedisplayed in region A05, terminal apparatus 11A transmits, to terminalapparatus 11B, image identification data identifying the dragged anddropped image, region data indicating a region (e.g., a region with apredetermined size and shape centered about the touched position) in theimage that the request source user was touching when dragging anddropping the image, and message data indicating the message associatedwith the operation button at the drag-and-drop location, in associationwith each other. Note that, as the image identification data, forexample, data indicating the frame number of the image constituting themoving image represented by the moving image data can be used, but thereis no limitation to this.

When the request source user touches a “transmit” button displayed inthe rightmost portion of region A05, terminal apparatus 11A transmitsimage identification data identifying the image that was displayed inregion A03 when the touch operation was performed to terminal apparatus11B. Also, when the request source user drags and drops the imagedisplayed in region A03 to the “transmit” button in region A05, terminalapparatus 11A transmits the image identification data identifying thedragged and dropped image and region data indicating the region in theimage touched by the request source user when dragging and dropping theimage in associated with each other to terminal apparatus 11B.

In region A06 of the request source screen, text box P04 and the likefor inputting a custom message that the request source user wants tosend to the request destination user is displayed. By uttering a messageto terminal apparatus 11A after touching a “Voice” button in region A06,the request source user can input text indicating the uttered message intext box P04. At that time, terminal apparatus 11A transmits a requestincluding the voice data acquired by voice acquisition means 118 tovoice recognition server apparatus 16, and as a response, displays thecontent of the text data transmitted from voice recognition serverapparatus 16 in text box P04.

Also, when the request source user performs a touch operation on a “Key”button in region A06, terminal apparatus 11A displays a virtualkeyboard. The request source user can input a message in text box P04 byperforming a touch operation on the displayed virtual keyboard.

When the request source user inputs a message in text box P04 and thenperforms a touch operation on the “Send” button in region A06, terminalapparatus 11A transmits message data indicating the message input intext box P04 to terminal apparatus 11B. Also, when the request sourceuser drags and drops the image displayed in region A03 to the “Send”button in region A06 or text box P04, terminal apparatus 11A transmitsthe image identification data identifying the dragged and dropped image,the region data indicating the region in the image touched by therequest source user when dragging and dropping the image, and themessage data indicating the message input in text box P04 in associationwith each other to terminal apparatus 11B.

In region A07 of the request source screen, a virtual operation buttonfor the request source user to instruct the direction of movement andthe like to the request destination user is displayed. The up-arrowbutton in region A07 is associated with message data indicating themessage “Please move forward”, and the down arrow button is associatedwith message data indicating the message “Please move backward”. Also,the right rotation arrow in region A07 is associated with message dataindicating the message “Please rotate to the right”, and the leftrotation arrow is associated with message data indicating the message“Please rotate to the left”. When the request source user performs atouch operation on any of these operation buttons, terminal apparatus11A transmits message data associated with the operation button on whichthe touch operation was performed to terminal apparatus 11B.

In region A08 of the request source screen, a map showing the currentposition of the request destination user (terminal apparatus 11B) isdisplayed. For this reason, terminal apparatus 11A receives the positiondata indicating the current position of terminal apparatus 11B, which istransmitted from terminal apparatus 11B at predetermined time intervals.Terminal apparatus 11A transmits a request including the position datareceived from terminal apparatus 11B to route search server apparatus13, and displays the map in region A08 using the map data transmittedfrom route search server apparatus 13 as a response. Terminal apparatus11A adds a circle mark indicating the current position of the requestdestination user indicated by the most recent position data transmittedfrom terminal apparatus 11B on the map, specifies a movement directionof the request destination user based on the change over time in theposition of the request destination user indicated by the position datatransmitted from terminal apparatus 11B, adds an arrow indicating thespecified movement direction on the map, and displays the result inregion A08.

In region A09 of the request source screen, a radio button for therequest source user to select whether to convey his or her currentposition to the request destination user “ambiguously” or “distinctly”is displayed. When the request source user selects “ambiguous” in regionA09, terminal apparatus 11A transmits position data that ambiguouslyindicates the current position of terminal apparatus 11A to the terminalapparatus 11B at predetermined time intervals. On the other hand, whenthe request source user selects “distinct” in region A09, terminalapparatus 11A transmits position data distinctly indicating the currentposition of terminal apparatus 11A to terminal apparatus 11B atpredetermined time intervals. Here, the position data that ambiguouslyindicates the current position is, for example, data that indicates thecurrent position with an accuracy on the order of several hundredmeters. By contrast, the position data that distinctly indicates thecurrent position is, for example, data that indicates the currentposition with an accuracy on the order of several meters.

In region A10 of the request source screen, a list box for conveying thecurrent state of the request source user to the request destination useris displayed. In the list box on the upper side of region A10, aplurality of icons indicating moods such as “happy”, “sad”, and “angry”are listed as options. In the list box on the lower side of region A10,a plurality of icons indicating states of physical strength, such as “Ihave a lot of physical strength left”, “I have some physical strengthleft”, “I don't have much physical strength left”, and “I have no morephysical strength” are listed as options. When the request source userchanges the icon indicating his or her current state in these listboxes, terminal apparatus 11A sends icon identification data identifyingthe changed icon to terminal apparatus 11B.

In region A11 of the request source screen, an icon indicating thecurrent state of the request destination user is displayed. Similarly tothe request source user, the request destination user can change theicons indicating his or her current state in a list box displayed on thescreen of terminal apparatus 11B. When those icons are changed, the iconidentification data identifying the changed icon is transmitted fromterminal apparatus 11B to terminal apparatus 11A. Terminal apparatus 11Adisplays the icon identified by the icon identification data transmittedfrom terminal apparatus 11B in this manner in region A11.

FIG. 5 is a diagram illustrating a screen displayed by terminalapparatus 11B (hereinafter referred to as a “request destinationscreen”). The request destination screen includes regions for displayingdisplay objects that are the same as or similar to those of theabove-described request source screen (FIG. 4 ). In FIG. 5 , regions fordisplaying display objects that are the same as or similar to those ofthe request source screen, which are included in the request destinationscreen, are denoted by reference numerals with the same last two digitsthereas.

In region B01 of the request destination screen, the moving imagerepresented by the moving image data transmitted from terminal apparatus11A to terminal apparatus 11B is displayed substantially in real time.

In the text box in region B02 of the request destination screen, anEnglish sentence indicated by text data received from translation serverapparatus 14 as a response obtained by terminal apparatus 11Btransmitting a request including message data received from terminalapparatus 11A to translation server apparatus 14 is displayed.

Similarly to when a sentence is displayed in the text box in region A02of the request source screen, when a sentence is displayed in the textbox in region B02 of the request destination screen, a voice utteringthe sentence is generated from sound generation apparatus 1102 ofterminal apparatus 11B. The request destination user can make the soundgeneration apparatus 1102 repeatedly generate the voice by performing atouch operation on the “Playback” button in region B02.

A plurality of virtual operation buttons are displayed in region B05 ofthe request destination screen. These operation buttons are operationbuttons for the request destination user to convey fixed messages to therequest source user, and a different message is associated with eachoperation button. The operation buttons displayed in region B05 can becustomized by the request destination user in the same manner as theoperation buttons (excluding the “Send” button) displayed in region A05of the request source screen.

When the request destination user performs a touch operation on any ofthe operation buttons displayed in region B05, terminal apparatus 11Btransmits message data indicating a message associated with theoperation button on which the touch operation was performed to terminalapparatus 11A.

In region B06 of the request destination screen, a text box or the likefor inputting a custom message that the request destination user wantsto send to the request source user is displayed. The role of the displayobjects included in region B06 of the request destination screen is thesame as the role of the display objects included in region A06 of therequest source screen except that the image cannot be dragged anddropped onto the text box in region B06, and therefore descriptionthereof will be omitted.

In region B08 of the request destination screen, a map showing thecurrent position of the request source user (terminal apparatus 11A) isdisplayed. For this reason, terminal apparatus 11B receives positiondata indicating the current position of terminal apparatus 11A, which istransmitted from terminal apparatus 11A at predetermined time intervals.Terminal apparatus 11B transmits a request including the position datareceived from terminal apparatus 11A to route search server apparatus13, and displays a map in region B08 using map data transmitted fromroute search server apparatus 13 as a response. Since the method bywhich terminal apparatus 11B generates the image of the map displayed inregion B08 is the same as the method by which terminal apparatus 11Agenerates the image of the map displayed in region A08, descriptionthereof will be omitted.

In region B09 of the request destination screen, a radio button for therequest destination user to select whether to “ambiguously” or“distinctly” convey the current position of the request destination userto the request source user is displayed. Since the role of the radiobutton included in region B09 of the request destination screen is thesame as the role of the radio button included in region A09 of therequest source screen, description thereof will be omitted.

In region B10 of the request destination screen, a list box forconveying the current state of the request destination user to therequest source user is displayed. Since the role of the list boxincluded in region B10 of the request destination screen is the same asthe role of the list box included in region A10 of the request sourcescreen, description thereof will be omitted.

In region B11 of the request destination screen, an icon indicating thecurrent state of the request source user is displayed. Since the role ofthe icon included in region B11 of the request destination screen is thesame as the role of the icon included in region A11 of the requestsource screen, description thereof will be omitted.

In region B12 of the request destination screen, the image identified bythe image identification data transmitted from terminal apparatus 11A toterminal apparatus 11B is displayed. That is, the image displayed inregion B12 is an image (still image) selected from the moving image bythe request source user. If the region data is associated with the imageidentification data received by terminal apparatus 11B from terminalapparatus 11A, terminal apparatus 11B displays, in region B12, an imageobtained by adding a display object (the broken-line circle in FIG. 5 )indicating the region indicated by the region data to the imageidentified by the image identification data.

In the text box in region B13 of the request destination screen, if themessage data is associated with the image identification data receivedby terminal apparatus 11B from terminal apparatus 11A, an Englishsentence indicated by the text data received from translation serverapparatus 14 as a response to terminal apparatus 11B transmitting arequest including the message data to translation server apparatus 14 isdisplayed. When a message is displayed in the text box in region B13,terminal apparatus 11B generates audio of the message. The requestdestination user can repeatedly listen to the audio by performing atouch operation on the “Playback” button in region B13.

In region B14 of the request destination screen, a map showing themovement route from the current position of the request destination user(terminal apparatus 11B) to the image capture position of the imagedisplayed in region B12 and the image capture direction of the image isdisplayed. Terminal apparatus 11B uses the position data last acquiredby position acquisition means 119 as the position data of the departurelocation, transmits a request including position data acquired byposition acquisition means 119 at the image capture time of the imagedisplayed in region B12 as the position data of the destination to routesearch server apparatus 13, and receives map data and movement routedata transmitted from route search server apparatus 13 as a response.Terminal apparatus 11B generates an image obtained by adding a brokenline indicating the movement route indicated by the movement route datareceived from route search server apparatus 13, a circle mark indicatingthe current position of the request destination user, an arrowindicating the movement direction of the request destination user, and afan mark indicating the image capture direction of the image displayedin region B12 to the map represented by the map data received from routesearch server apparatus 13, and displays the generated image in regionB12.

If the request destination user receives a request to buy an objectappearing in the image displayed in region B12 (e.g., the objectappearing in the region indicated by the broken-line circle in regionB12) from the request source user, for example, the request destinationuser can easily reach the location where the object instructed by therequest source user is located by moving according to the movement routedisplayed in region B14. Also, the request destination user can easilyspecify the object instructed by the request source user while viewingthe image displayed in region B12 at the reached location.

According to system 1 according to the above-described embodiment, therequest source user can easily convey a target object that the requestsource user wants the request destination user to perform some action onto the request destination user by presenting an image in the movingimage sent from the request destination user to the request destinationuser.

Also, according to system 1, the request source user can quickly andclearly convey a message to the request destination user by operatingthe operation buttons.

Also, according to system 1, even if the request source user and therequest destination user use different languages, the request sourceuser and the request destination user can communicate with each other.

Also, according to system 1, the request source user can easily confirmthe current position and the current movement direction of the requestdestination user, and therefore the request source user can easilyinstruct the direction in which he or she wants the request destinationuser to move next to the request destination user.

Also, according to system 1, the request source user can easily send animage and a message in association with each other to the requestdestination user by performing a simple operation such as dragging anddropping an image selected from a moving image onto an operation buttonassociated with a message or a text box in which a message has beeninput. Also, when the request source user drags and drops the image ontothe operation button or the like, the request source user can easilyconvey the target object to the request destination user by touching thetarget object appearing in the image.

MODIFIED EXAMPLES

System 1 described above may be modified in various ways within thescope of the technical idea of the present invention. Examples ofmodification are shown below. Two or more of the following modifiedexamples may be combined as appropriate.

(1) Instead of the image identification data of the image selected fromthe moving image by the request source user, image data representing theimage may be transmitted from terminal apparatus 11A to terminalapparatus 11B. In this case, terminal apparatus 11B displays the imagerepresented by the image data received from terminal apparatus 11A inregion B12 of the request destination screen.

(2) In the above-described embodiment, terminal apparatus 11A stores atleast a portion of the moving image data received from terminalapparatus 11B, which corresponds to the most recent past predeterminedtime. The portion of the moving image data received by terminalapparatus 11A from terminal apparatus 11B that needs to be stored may bea portion corresponding to a predetermined amount, and does notnecessarily need to be a portion corresponding to a predetermined time.For example, terminal apparatus 11A may store at least a portion of themoving image data received from terminal apparatus 11B, whichcorresponds to the most recent past predetermined data amount.

(3) Terminal apparatus 11A may display, on the image in region A03, theresult of recognizing an object appearing in an image that is selectedby the request source user and displayed in region A03 of the requestsource screen. In this case, terminal apparatus 11A is provided with animage recognition means for recognizing an object appearing in the imagedisplayed in region A03 using a known image recognition technique, anddisplays an image obtained by adding a display object indicating theresult of recognition performed by the image recognition means in regionA03. Also, processing for image recognition may be performed by, forexample, a server apparatus or the like that provides an imagerecognition service connected to system 1.

(4) Instead of transmitting and receiving message data indicating amessage between terminal apparatus 11A and terminal apparatus 11B,message identification data for identifying a message may be transmittedand received. In this case, terminal apparatus 11A and terminalapparatus 11B store message data indicating fixed messages in advance inassociation with message identification data used in common betweenterminal apparatus 11A and terminal apparatus 11B. For example, terminalapparatus 11A transmits the message identification data of the messagedesignated by the request source user to terminal apparatus 11B, andterminal apparatus 11B displays the message indicated by the messagedata identified by the message identification data received fromterminal apparatus 11A. Similarly, terminal apparatus 11B transmitsmessage identification data of a message designated by the requestdestination user to terminal apparatus 11A, and terminal apparatus 11Adisplays the message indicated by the message data identified by themessage identification data received from terminal apparatus 11B.

(5) In the above-described embodiment, terminal apparatus 11A andterminal apparatus 11B receive message data indicating a sentence fromthe partner apparatus, and acquire voice data representing a voiceuttering the sentence indicated by the message data from voice synthesisserver apparatus 15. Instead of or in addition to this, voice datarepresenting the sound of uttering the sentence of the message may betransmitted and received as message data between terminal apparatus 11Aand terminal apparatus 11B.

(6) In the above-described embodiment, when the request destination useris an apparatus such as a robot provided with an operation means forperforming a physical operation, operation instruction data forinstructing a physical operation to the request destination user who isan apparatus may be transmitted as message data from terminal apparatus11A to terminal apparatus 11B. In this case, the request destinationuser who is an apparatus operates according to the operation instructionindicated by the message data. Also, if the request destination user isan apparatus such as a robot provided with a movement means for moving,the request destination user who is an apparatus may move according tothe movement route indicated by the movement route data.

(7) In the above-described embodiment, at least some of the requestsmade by terminal apparatus 11A to route search server apparatus 13,translation server apparatus 14, and voice synthesis server apparatus 15may be made by terminal apparatus 11B. In this case, the data acquiredby terminal apparatus 11B from those server apparatuses may betransmitted to terminal apparatus 11A. Similarly, in the above-describedembodiment, at least some of the requests made by terminal apparatus 11Bto route search server apparatus 13, translation server apparatus 14,and voice synthesis server apparatus 15 may be made by terminalapparatus 11A. In this case, the data acquired by terminal apparatus 11Afrom those server apparatuses may be transmitted to terminal apparatus11B.

(8) In the above-described embodiment, at least some of the processingperformed by route search server apparatus 13, translation serverapparatus 14, voice synthesis server apparatus 15, and voice recognitionserver apparatus 16 may be performed by terminal apparatus 11A orterminal apparatus 11B.

(9) In the above-described embodiment, the operation buttons that aredisplayed in region A07 of the request source screen (FIG. 4 ) areexamples, and operation buttons for instructing other operations mayalso be displayed in region A07. For example, an operation buttonassociated with a message such as “Please jump” and an operation buttonassociated with a message such as “Please move forward quickly” or“Please move forward slowly” may be displayed in region A07.

(10) In the description of the above-described embodiment, the requestsource user and the request destination user are assumed to be oneperson each, but at least one of the request source user and the requestdestination user may be a plurality of people. In this case, since thenumber of terminal apparatuses that simultaneously perform datacommunication in order to perform the local proxy service is three ormore, management server apparatus 12 may relay the data communicationbetween those terminal apparatuses.

Also, if there are a plurality of request source users, it isinconvenient if contradicting instructions are given to the requestdestination user at the same time from the plurality of request sourceusers. In order to avoid such an inconvenience, only one of theplurality of terminal apparatuses 11A may have instruction authority,and the transmission of image identification data and message data fromterminal apparatus 11A having no instruction authority to terminalapparatus 11B may be restricted.

(11) In the above-described embodiment, the icon selected in region A10of the request source screen (FIG. 4 ) is selected by the request sourceuser. Instead of this, for example, terminal apparatus 11A may estimatethe state of the request source user based on an image of the face ofthe request source user captured by terminal apparatus 11A, the voice ofthe request source user picked up by terminal apparatus 11A, and thelike, and terminal apparatus 11A may automatically select the icon inregion A10 according to the estimation result. Similarly, terminalapparatus 11B may estimate the state of the request destination userbased on an image of the face of the request destination user capturedby terminal apparatus 11B, the voice of the request destination userpicked up by terminal apparatus 11B, and the like, and terminalapparatus 11B may automatically select the icon in region B10 of therequest destination screen (FIG. 5 ) according to the estimation result.

Note that the type of user's state displayed in region A10 or region B10is not limited to mood and physical strength.

(12) Using system 1, the request source user may request the requestdestination user to make an electronic payment using an image code suchas a QR code (registered trademark) or a barcode at the local site. Forexample, terminal apparatus 11A can transmit an image code forelectronic payment of the request source user to terminal apparatus 11B,and a store can receive payment of a fee from the request source userdue to a staff member or the like of the store at the local site readingthe image code displayed on terminal apparatus 11B with a code reader.

Also, terminal apparatus 11B can transmit an image obtained by capturingan image of the image code for electronic payment of the store at thelocal site to terminal apparatus 11A, and the store can receive paymentof money from the request source user due to the request source userperforming an operation of paying the designated balance to the storeidentified by the image code displayed on terminal apparatus 11A. Inthis case, the store confirms that the processing of payment of thedesignated fee by the request source user is complete using an imagetransmitted from terminal apparatus 11A to terminal apparatus 11B.Accordingly, in order to prevent the image transmitted from terminalapparatus 11A to terminal apparatus 11B from being forged, for example,terminal apparatus 11B determines whether or not the time displayed onthe payment screen substantially matches the current time, and if theydo not substantially match, a warning may be displayed.

(13) In the above-described embodiment, if the request source user wantsto present any image in the moving image to the request destinationuser, it is necessary to temporarily display the image in region A03 ofthe request source screen. The method for designating the image that therequest source user wants to present to the request destination user isnot limited to this. For example, by performing a predeterminedoperation such as a double tap on region A01, the image identificationdata of the image may be transmitted from terminal apparatus 11A toterminal apparatus 11B without temporarily displaying the image inregion A03. Also, when the transmission source user performs anoperation such as dragging and dropping from region A01 to the “Send”button or the like in region A05, the image identification data of theimage displayed in the moving image may be transmitted from terminalapparatus 11A to terminal apparatus 11B.

(14) In the above-described embodiment, the request source user canaccumulate candidate images in region A04 by performing an intentionaloperation such as dragging and dropping an image from region A03 of therequest source screen to region A04. Instead of this, all of the imagesdisplayed in region A03 by the request source user may be accumulated inregion A04. In this case, automatically-accumulated images andintentionally-accumulated images may be distinguished from each other byadding bookmarks to the images intentionally accumulated in region A04by the request source user, or the like.

(15) If the request destination user is at a location where imagecapture for streaming is allowed but recording is prohibited, recordingof moving images (including recording of images constituting the movingimages) in terminal apparatus 11A and terminal apparatus 11B may be madeimpossible to execute. In this case, for example, management serverapparatus 11 or the like need only store a database storing prohibitedarea data indicating areas where recording is prohibited, prohibitionperiod data indicating a period during which recording is prohibited,and permission condition data indicating a condition of an authorizedperson who is exceptionally permitted to record (e.g., a host of anevent where recording is prohibited), and terminal apparatus 11A andterminal apparatus 11B need only determine whether or not recording ofmoving images is prohibited by referring to the database. Then, if therequest source user or the request destination user is an authorizedperson who is exceptionally permitted to record, confirmation that he orshe is the authorized person (e.g., inputting a password known only tothe authorized person, etc.) is performed, and if it is confirmed thathe or she is an authorized person, terminal apparatus 11A and terminalapparatus 11B may be able to record a moving image. Note that, forexample, a method of adding a code for prohibiting recording to themoving image data generated in terminal apparatus 11B and thereaftertransmitting the data to terminal apparatus 11A is conceivable as amethod for prohibiting recording of a moving image.

(16) In the above-described embodiment, the request destination user canconfirm the movement route from the current position to the imagecapture position of the image presented by the request source user onthe map. Instead of or in addition to this, terminal apparatus 11B maydisplay the movement route to the request destination user using AR(Augmented Reality) technology. In this case, when the requestdestination user captures an image of the surrounding area using camera107 of terminal apparatus 11B, display objects such as arrows indicatingthe movement route are displayed in a superimposed manner together withthe image captured by camera 107 on touch screen 104 of terminalapparatus 11B. As a result, the request destination user can reach thedestination more intuitively.

1. A terminal apparatus comprising a processor configured to execute: animage data receiving step for receiving moving image data transmitted inreal time from a communication partner apparatus; a storing step forstoring at least a portion of moving image data received in the imagedata receiving step, the portion corresponding to a most recent pastpredetermined amount; an operation receiving step for receive a useroperation on an image being displayed on a display apparatus; a displayinstructing step for displaying a moving image represented by movingimage data received in the image data receiving step on the displayapparatus in real time, and displaying, on the display apparatus, one ormore images designated by a user operation received in the operationreceiving step among images constituting a past moving image representedby moving image data stored in the storing step, a selected imageacquiring step for acquiring image identification data for identifying aselected image among a plurality of images constituting a moving imagerepresented by the moving image data received in the image datareceiving step; and transmitting step for transmitting imageidentification data acquired in the selected image acquiring step, orimage data representing the image identified by the image identificationdata to the communication partner apparatus, wherein the processorconfigured to execute, in the selected image acquiring step, acquiringimage identification data for identifying an image that was displayed bythe display apparatus when a user operation on moving image displayed bythe display apparatus was received in the operation receiving step, asimage identification data of a selected image or image identificationdata of a selected candidate image, and in the selected image acquiringstep, acquiring image identification data for identifying an image thatwas designated by a user operation received in the operation receivingstep, the user operation being made for selecting an image from amongone or more images designated by a user operation and displayed by thedisplay apparatus, as image identification data of a selected image orimage identification data of a selected candidate image.
 2. (canceled)3. (canceled)
 4. The terminal apparatus according to claim 1, whereinthe processor configured to execute, if image identification data of aselected candidate image was acquired in the selected image acquiringstep, displaying an image identified by the image identification data onthe display apparatus as a selected candidate image in the displayinstructing step, and if a user operation for selecting an image fromamong one or more candidate images displayed by the display apparatuswas made in the operation receiving step, acquiring image identificationdata of the selected image as selected image identification data in theselected image acquiring step.
 5. The terminal apparatus according toclaim 1, wherein the processor configured to execute, in the operationreceiving step, receiving a user operation for designating any regionincluded in the selected image identified by the image identificationdata acquired in the selected image acquiring step, and in thetransmitting step, transmitting region data indicating the regiondesignated by the user to the communication partner apparatus.
 6. Theterminal apparatus according to claim 1, wherein the processorconfigured to execute, an image recognizing step for recognizing anobject appearing in the selected image identified by the imageidentification data acquired in the selected image acquiring step, andin the display instructing step, displaying on the display apparatus, animage obtained by adding a display object indicating a result ofrecognition performed in the image recognizing step to a selected imageidentified by image identification data acquired in the selected imageacquiring step.
 7. The terminal apparatus according to claim 1, whereinthe processor configured to execute, in the display instructing step,displaying a virtual operation button on the display apparatus, and if auser operation on the virtual operation button was received in theoperation receiving step, transmitting message identification data foridentifying a message corresponding to the virtual operation button ormessage data indicating the message, to the communication partnerapparatus, in the transmitting step.
 8. The terminal apparatus accordingto claim 7, wherein the processor configured to execute, if a useroperation of associating an image selected by the user with the virtualoperation button was received in the operation receiving step,transmitting message identification data for identifying a messagecorresponding to the virtual operation button or message data indicatingthe message, in association with image identification data acquired inthe selected image acquiring step or image data representing the imageidentified by the image identification data, to the communicationpartner apparatus, in the transmitting step.
 9. The terminal apparatusaccording to claim 7, wherein the message data is text data indicating asentence written in a language used by a user of the communicationpartner apparatus or voice data representing a sound of uttering thesentence.
 10. The terminal apparatus according to claim 7, wherein themessage data is data indicating a state of the user.
 11. The terminalapparatus according to claim 7, wherein the message data is operationinstruction data for instructing a physical operation to thecommunication partner apparatus. 12-21. (canceled)